Abstract Title Page 

Not included in page count. 



Title: 

The Statistical Power of the Cluster Randomized Block Design with Matched Pairs 
Simulation Study 



Author(s): 

Nianbo Dong, Ph.D. and Mark Lipsey, Ph.D. 
Vanderbilt University 



2010 SREE Conference Abstract Template 




Abstract Body 

Limit 5 pages single spaced. 

Background/context: 

Group-randomized experiments, in which groups (clusters) are randomly assigned to the 
treatment and control conditions, are widely used in educational research (e.g., Borman, Slavin, 
Cheung, Chamberlain, and Madden, et al., 2005; Cook, Hunt, & Murphy, 2000). In designing 
such studies, it is important to have sufficient power to detect the effect of the intervention. 
Raudenbush, Martinez, & Spybrook (2007) summarized five factors associated with the power in 
group-randomized experiments: (1) the size of the true average effect size (ES) of the 
intervention, (2) the intra-class correlation (ICC) indicating the fraction of the variance that lies 
between clusters, (3) the number of clusters studied (7), (4) the number of individuals per cluster 
(n), and (5) the adopted level of statistical significance ( a ) for either a one-tailed or two-tailed 
test. 



In addition, using covariance adjustment can improve precision (Bloom, 2006). The 
bigger the R 2 for the relationship between a covariate and the dependent variable, the smaller 
the minimum detectable effect size (MDES) is. This is a commonly used approach in group- 
randomized studies (e.g., Borman et al, 2005). 

Furthermore, pre-randomization blocking or matching is another important approach that 
can control background variability and increase power for detecting the effects of treatment 
(Hedges, 2007; Raudenbush, Martinez, & Spybrook, 2007). One application of this approach is 
the randomized matched-pairs design, one variant of the randomized block design, i.e., first 
matching clusters (e.g., schools) into pairs on a covariate, then randomly assigning the clusters in 
each pair to two conditions. Matching can improve face validity, and improve precision in some 
circumstances (Bloom, 2007; Raudenbush, Martinez, & Spybrook, 2007). In addition to the 
factors listed above, two additional factors are also associated with the power in the matched- 
pairs (MP) randomized-block design, i.e., the within-pair correlation and the true effect size 
variability (ESV) across pairs (Raudenbush, Martinez, & Spybrook, 2007). 

Pairing will improve statistical power when “the gain in predictive power outweighs the 
loss of degrees of freedom” associated with the blocked pairs (Bloom, 2007). Bloom (2007) 
derived a formula to calculate the minimum incremental R 2 that will improve power, and 
illustrated the required R 2 for varying numbers of clusters. Furthermore, Raudenbush, Martinez, 
& Spybrook (2007) compared power for the completely randomized cluster design with and 
without covariance adjustment, and the matched-pair (MP) randomized-block design for 
different parameter combinations. Their conclusions are consistent with Bloom (2007) that 
matching will not improve power unless the percentage of the variance explained by matching 
reaches a threshold. 

These papers provided a good theoretical framework and mathematical formulation for 
determining the threshold above which matching will improve power. However, as Raudenbush, 
Martinez, & Spybrook (2007) noted “there is no closed-form mathematical expression for 
determining the variance explained by matching on W when W randomly varies within pairs” 
(p.21). In practice it is unclear how large the incremental R 1 of matching can be under various 
situations. Furthermore, power was not directly compared between the models with random and 
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fixed pair effects with covariance adjustment and the model for the group randomized design 
with covariance adjustment. 

Purpose / objective / research question / focus of study: 

This study uses simulation techniques to examine the statistical power of the group- 
randomized design and the matched-pair (MP) randomized block design under various parameter 
combinations. Both nearest neighbor matching and random matching are used for the MP design. 
The power of each design for any parameter combination was calculated from 1,000 simulated 
datasets. The random pair effect HLM model with and without covariance adjustment, and the 
fixed pair effect HLM model with and without covariance adjustment were used to analyze the 
data from the matched-pair design, and the HLM with covariance adjustment was used to 
analyze the data from the completely randomized cluster design. 

Setting: 

Simulated setting, in which the individuals are nested within clusters, e.g., students are 
nested within schools. 

Population / Participants / Subjects: 

Samples were generated for the simulation using statistical models with various 
combinations of parameters. 

Intervention / Program / Practice: 

Simulated interventions with various effect sizes and effect size variability across pairs/. 

Research Design: 

Data were generated using various parameters: (1) the number of clusters ( J = 20, 40, and 
60), (2) the number of individuals per cluster ( n = 20), (3) the intra-class correlation (ICC = .1 
and .2), (4) Rf (= .2 and .5) for individual level covariate, (5) Rf (=R{ = .2 and .5) for cluster 
level covariate, (6) the effect size (ES = .2 and .5), and (7) the effect size variability (ESV = 0, 
and .01) across pairs. The number of total parameter combinations is 48. 

Both nearest neighbor matching and random matching were used for the matched-pair 
(MP) group-randomized design. The nearest neighbor matching procedure rank orders clusters 
by a cluster-level covariate (W) and pairs adjoining clusters. In random matching, clusters are 
paired randomly. The results from random matching provide information about how much the 
quality of matching matters. 

Treatment effects were estimated for the group-randomized design using a two-level 
HLM model with covariance adjustment. They were estimated for the nearest neighbor and 
random matching using four models: random pair effects with and without covariance adjustment, 
and fixed pair effects with and without covariance adjustment. The p- value, the effect size, the 
i? 2 for the individual level covariate, Rf for the cluster level covariate, the unconditional and 
conditional ICC etc. were calculated for each model with each simulated data set. 
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For each of 48 parameter combinations, 1,000 datasets were generated and analyzed 
using SAS. The percentage of the studies whose 79-value were less than .05, two-tailed, estimated 
the power. The average ES, Rf , R 2 , and ICC etc. are calculated. 

Data Collection and Analysis: 

Data Generation 

Using the parameters specified above, data were generated following two stages below. 

1. Generating a master dataset satisfying the models in Appendix C. 

The outcome variable, K. , was created satisfying; (1) the specified unconditional ICC, 

which was satisfied by varying level- 1 variance and level-2 variance, (2) an individual level 
covariate, X , and a cluster-level covariate, W, which were normally distributed, and their 
variance explained at level 1 and level 2 were Rf and R 2 , respectively. The overall pooled 

standard deviation ( y'r 0 + a 2 ) of Y.. was calculated. 

2. Adding treatment effect on T. . 

Using the same master dataset generated above, clusters were randomly assigned to two 
conditions, clusters in the nearest neighbor matched pairs were randomly assigned to two 
conditions, and clusters in the randomly matched pairs were randomly assigned to two conditions 

to produce three study designs. The same treatment effect, (ES + v) * ^]r 0 + a 2 , where 
v~ N(0, 4ERSV) was a random term indicating that the treatment effect varied across pairs, was 
added to the K. for the treatment group to produce final dependent variable, Y'„ . 

Data Analysis 

A two-level HLM model with covariance adjustment was used to estimate the treatment 
effect for the data from completely group-randomized design. 

Level 1 (individual) 

(1) li-/8 oy +/S v C,+r, r v ~ N(0,cr 2 ) 

Level 2 (cluster) 

(2) Poj = Y 00 + Yoi W j + YoiTREATMENT; + /u Q V 0 ~ N ( 0,r) 

(3) Pxj = 7,o 

The reduced model is: 

(4) Xj = 7oo + Yoi W i + Y 02 TREATMENJ + y l0 + £ 

As mentioned in Research Design, four models were used to estimate the treatment effect 
for the matched-pair group-randomized design. The random pair effect HLM model with 
covariance adjustment is presented below: 
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Where k is the third level, pair. 



The fixed pair effect HLM model with covariance adjustment is presented below: 

E,jk = Yooo + Yo\qW jk + Y 020TREATMEN Tj k + y l00 X ijk + ^ y 00 m P n 

m 

+ Pojk + r ijk 



ijk 



■N( 0 ,a'),v 0 jk ~N( 0 ,T£>) 



Where k is the third level, pair. P m is a dummy variable indicating pair m. 

The random and fixed pair effect HLM models without covariance adjustment are similar 
with Models (5) and (6), respectively except that no covariate variables (X and Wj are included. 

Findings / Results: 

Partial results are reported below although the overall simulation and analyses have not 
been completed. Table 1 presents the simulation results (with 1,000 replications) for the 
parameter combination: J = 40, n = 20, ICC = .1, ES = .3, Rf = R% = .5, and ESV = 0 and .01, 
respectively. First, notice that the power of the group-randomized design under this parameter 
combination for ESV = 0 is .93. The Optimal Design software (Liu, Spybrook, Congdon, 
Martinez, & Raudenbush, 2006) produced a power of .84 for the same parameter combination 
but did not take account of R , 2 (Figure 1). Based on Bloom’s (2006, p. 17) Minimum Detectable 
Effect Size (MDES) formula, the power associated with this parameter combination is .93, and 
the power ignoring Rf is .85. Hence this simulation result is consistent with the results derived 
from Bloom’s mathematical formulation. Second, the group-randomized design had the largest 
power, but the MP randomized block design, when analyzed with covariance-adjustment, had 
almost identical power. In addition, there was no difference between the random and fixed pair 
effects MP models or between the two matching procedures for models with covariance- 
adjustment. However, for the models without covariance-adjustment, nearest neighbor matching 
had more power than random matching (.81 vs. .72). Third, for the MP randomized block design, 
the analytic models with covariance-adjustment had more power than those without covariance- 
adjustment. Fourth, the power with ESV (effect size variability) = 0 was slightly higher than the 
power with ESV = .01 when the models with covariance-adjustment were used, but did not vary 
much for the models without covariance adjustment. 

Table 2 presents the simulation results with the same parameter combination in Table 1 
except with Rf = R* = .2. The results have the same pattern as in Table 1 but power is lower 
than in Table 1. 
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Table 3 presents the simulation results for J = 20, n = 20, ICC = .1, ES = .3, Rf = R 2 = 
.2 and 5, and ESV = 0 and .01, respectively for the models with covariance-adjustment. The 
group-randomized design still had the largest power. The random pair effects models produced 
similar power, but the fixed pair effects models produce slightly smaller power. 

Table 4 presents the average estimated R 2 of various study designs. First, notice that the 
simulated data were as designed in terms of R 2 . Next, with covariance-adjustment for ESV = .1, 
Rf from the random pair effects model was a little bigger than the other models. However, the 
overall R 2 did not vary across study designs and analytic models when covariance was adjusted. 

Conclusions: 

Under the parameter combination illustrated, the simulation showed that the group- 
randomized design had the greatest power. By including the covariate in the model, the power 
was improved, and the quality of the matching had only a small effect on power. However, when 
R 2 was big, but the covariate was omitted, the nearest neighbor matching had more power than 
random matching. In addition, when the sample size was small, the random pair effects model 
produced bigger power than the fixed pair model, which seems to be against the current theory 
and needs further exploration. Finally, when covariance was adjusted, the incremental R 2 of 
matching was limited. 
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Appendix B. Tables and Figures 
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Table 1 

Powers of various study designs from simulation' (J=40, n=20, ICC=.l, ES=.3, 
Rf = Rj =.5, a = .05 for two-tailed test ) 







With 


Without 




Analytic 

Model 


covariance- 


covariance- 


Study Design 


adjustment 


adjustment 






ESV* 


ESV* 






0 .01 


0 .01 


Completely group- 
randomized design 


2-level HLM 


,93 e .90 






Random Pair 


.91 .89 


.81 .81 


Matched-pair Randomized 
Block Design 


Effects Model 






(Nearest neighbor matching) 


Fixed Pair 
Effects Model 


.90 .88 


.81 .81 




Random Pair 


.91 .88 


.72 .72 


Matched-pair Randomized 
Block Design 


Effects Model 






(Random matching) 


Fixed Pair 
Effects Model 


.90 .87 


.72 .71 



”1,000 replications. 

*ES V : Effect Size Variability across pair or cluster, represented by the variance of the 
Effect Size adopted. 



c Under this parameter combination, the Optimal Design software (v.1.77) produced a 
power of .84, which is lower than .93 because Optimal Design did not take account of Rf . 
Based on the formula on page 17 in Bloom (2006), the power associated with this 
parameter combination is .93, and the power ignoring Rf is .85. 
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Table 2 



Powers of various study designs from simulation' (J=40, n=20, ICC=.l, ES=.3, 
Rf = R-l =.2, a = .05 for two-tailed test) 



With Without 

covariance- covariance- 

Study Design A ^^j C adjustment adjustment 

ESV h ESV h 







0 


.01 


0 


.01 


Completely group- 
randomized design 


2-level HLM 


.76 


.73 








Random Pair 


.77 


.73 


.70 


.72 


Matched-pair Randomized 
Block Design 


Effects Model 










(Nearest neighbor matching) 


Fixed Pair 
Effects Model 


.74 


.71 


.70 


.72 




Random Pair 


.75 


.71 


.70 


.68 


Matched-pair Randomized 
Block Design 


Effects Model 










(Random matching) 


Fixed Pair 
Effects Model 


.73 


.70 


.70 


.68 



fl l,000 replications. 

*ES V : Effect Size Variability across pair or cluster, represented by the variance of the 
Effect Size adopted. 
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Table 3 



Powers of various study designs from simulation 1 (J=20, n=20, ICC=.l, ES=.3, a = .05 
for two-tailed test ) 







R?=R;=.2 


R? = R;=5 


Study Design 


Analytic 






Model 


ESV* 


ESV* 






0 .01 


0 .01 


Completely group- 
randomized design 


2-level HLM 


.44 .42 


.64 .61 




Random Pair 


.43 .38 


.61 .57 


Matched-pair Randomized 
Block Design 


Effects Model 






(Nearest neighbor matching) 


Fixed Pair 
Effects Model 


.38 .33 


.56 .53 




Random Pair 


.40 .37 


.56 .52 


Matched-pair Randomized 
Block Design 


Effects Model 






(Random matching) 


Fixed Pair 
Effects Model 


.37 .35 


.53 .52 



n l,000 replications. All models are with covariance-adjustment. 

*ES V : Effect Size Variability across pair or cluster, represented by the variance of the 
Effect Size adopted. 
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Table 4 



Average Estimated R 2 of various study designs from simulation' (J=40, n=20, ICC=.l, ES=.3, 
R 2 = R 2 =5) 









ESV'=0 




ESV fo =.01 




Study Design 


Analytic Model 


Total 






Total 










R 2 


Rl 


R 2 


R 2 


Rl 




Completely group-randomized 
design 


2-level HLM 


.50 


.48 


.50 


.50 


.45 


.50 




Random Pair 


.50 


,49 c 


.50 


.50 


,51 c 


.50 


Matched-pair Randomized 
Block Design 

(Nearest neighbor matching) 


Effects Model 

Fixed Pair Effects 
Model 














.50 


.49 


.50 


.50 


.45 


.50 




Random Pair 


.50 




.50 


.50 




.50 


Matched-pair Randomized 
Block Design 


Effects Model 


,48 c 


.50 c 










(Random matching) 


Fixed Pair Effects 
Model 


.50 


.48 


.50 


.50 


.44 


.50 



fl l,000 replications. All models are with covariance-adjustment. 

fo ESV: Effect Size Variability across pair or cluster, represented by the variance of the Effect 
Size adopted. 

^Representing the proportion of the sum variances at level 2 and level 3 that are explained by 
the covariate W. It was calculated for the purpose of comparison with the other models. 
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Figure 1. Power vs. Effect Size by Optimal Design 
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Appendix C. Models Used to Generate Data at Stage 1 

Not included in page count. 

Unconditional Model: 

(!) Y ij =Yoo +^o + l 1 o~ N(0 ,t 0 ) , r f j~ N(0 ,Oq) 

Where: 

L = Outcome for individual, i in cluster, j 
7 00 = Grand mean of the outcome 
/u 0 = Random error term for cluster, j 
r = Random error term for individual, i in cluster, j 

Unconditional ICC = — — — - 
T o + 

Pooled standard deviation: ^jr 0 + o\ 

Conditional Model: 

Level 1 (individual) 

( 2 ) + + 

Level 2 (cluster) 

(3) Poj — Yoo + r n Wj + Po 

(4) A j = /io 

The reduced model is: 

(5) ^ = 7oo + 7oi^ + Xio^ + + r ij 

Where: 

Yy = Outcome for individual, i in cluster, j 

Xy = Covariate for individual, i in cluster, j (normally distributed) 
W, = Cluster-level covariate for cluster, j (normally distributed) 

7 00 = Grand mean of the outcome 

7 01 = Coefficient of X . 

7 10 = Coefficient of C„ 

= Random error term for cluster, j 
r = Random error term for individual, i in cluster, j 

o 2 — o 2 

Individual level covariate R, 2 = — 5 — — — 

Cluster level covariate R 2 = — L 



r 9 ~N(0,o?) 

V 0 ~N( 0,Tj) 
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